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ABSTRACT 

It is still unclear on what and how tool and learner characteristics influence tool use and consequently 
performance in computer-based learning environments (CBLE’s). This study examines the relationships 
between tool-related characteristics (tool presentation: non-/embedded tool and instructional cues: non- 
/explained tool functionality) and learner-related characteristics (self-efficacy and goal orientation) as well as 
their effects on tool use (quantity and quality) and performance in a CBLE. One hundred and forty students, 
without statically difference in prior knowledge, were randomly assigned to the four conditions (Embedded and 
non-embedded with explained tool functionality and embedded and non-embedded with non-explained tool 
functionality) to study a hypertext. Results reveal that embedding tools influenced positively quantity of tool 
use and negatively quality of tool use. Partial effects of explained tool functionality were found. There were 
significant interactions of goal orientation (mastery avoidance) and condition on quality of tool use. 
Performance approach influenced quality of tool positively and self-efficacy influenced negatively quantity of 
tool use. Only quantity of tool use affected performance. The implications of these results for future research on 
tool use in CBLE’s are discussed. 

Keywords: adjunct questions; self-efficacy; goal orientation; tool presentation; instructional cues 

INTRODUCTION 

In computer-based learning environments (CBLE’s) support devices, also referred as tools, (Iiyoshi & Hannafm, 
1998; Viau & Larivee, 1993) are often implemented under the assumption that tool use can have a positive 
impact on learning outcomes/performance (Moos & Azevedo, 2009). Tool use is generally explored either 
quantitatively and/or qualitatively. The frequency in which learners access tools and the time they spend on the 
tools are commonly considered as indicators of quantity of tool use. On the other hand, when and how learners 
use the tools and whether the usage is in line with the instructional intentions of particular tools are considered 
as the quality of tool use. 

Jiang and Elen (2011) reported that the quality of tool use, specifically the use of adjunct questions (i.e., reading 
attention, reading sequence and the answers provided to the questions) accounts for the performance (post-test). 
Quantity of tool use (frequency of tool use and time on tool) showed mixed effects on performance. In a 
different study, Viau and Larivee (1993) examined the effects of the usage of two tools, namely a glossary and a 
navigation map, on performance. They found that the quantity of tool use (frequency of tool use and time on 
tool) were the best predictors of performance (post-test). Quality of tool use was not studied. 

While the clear effects of tool use on performance are not concrete, they give an indication that using tools 
appropriately can lead to better performance and therefore tool usage should not be taken for granted. However, 
positive effects of tool use on performance can only be expected, according to Perkins (1985), only if the 
tool(s) is present and functional, if learners recognize the tools and the tools’ functionality and if learners are 
motivated to use the tools and use the tools optimally. Unfortunately, learners tend to not use the tools and 
when they do, they do it suboptimally (e.g., Aleven, Stahl, Schworm, Fischer, & Wallace, 2003; Elen & Louw, 
2006; Perkins, 1985). This problematic tool use may related to the interaction between different learner and tool 
characteristics (see reviews: Aleven, et al., 2003; Clarebout & Elen, 2006) which is congruent with the theory 
of aptitude-treatment interaction (ATI) (Cronbach & Snow, 1977). The ATI points out that instructional 
strategies or treatments, namely tools interact with aptitudes, defined as any measurable characteristic of the 
learner (Regian & Shute, 1992), in this case learner-related characteristics. 

The present contribution will explore in a CBLE the relationship between different learner-related and tool- 
related variables on tool use and performance. These variables are described in detail as follows. 

LEARNER CHARACTERISTICS 

Prior knowledge has been identified as a learner characteristic affecting tool use and performance (Aleven, et 
al., 2003; Clarebout & Elen, 2006). Elen and Louw (2006) found that learners with higher prior knowledge had 
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better performance in the post-test. Renkl (2002) reported that low prior knowledge learners used tools more 
frequently than high prior knowledge learners. Iiyoshi and Hannafin (1998) suggested that prior knowledge may 
impact performance through different quantitative and qualitative aspects of tool use, for instance, the frequency 
of tools use. While prior knowledge seems to be an important characteristic regarding tool use, this study only 
controls for prior knowledge and focused on Perkins’s last condition on tool use (learners are motivated to use 
tools), hence learner motivational characteristics will be investigated. 

Motivational characteristics seem crucial to determine tool use (Perkins, 1985). Motivation is a broad term that 
derives in many associated constructs exploring different motivational aspects (Murphy & Alexander, 2000). 
Self-efficacy and goal orientation are two motivational characteristics that have caught the attention of research 
on tool use (Aleven, et al., 2003; Newman, 1994). Self-efficacy is defined as the beliefs about one’s capabilities 
to organize or perform the courses of action to produce a given achievement (Bandura, 1997). Goal orientation 
is defined as the set of beliefs that reflect the reasons and intentions to engage and achieve a task (Dweck, 
1986). 

Subcategories of goal orientation have been identified, mainly a mastery orientation (defined in intrapersonal 
terms) and a performance orientation (defined in normative terms) (Dweck & Legget, 1988), which at the same 
time are further subdivided and valenced into mastery avoidance and mastery approach as well as performance 
avoidance and performance approach, respectively (Elliot & McGregor, 2001). Mastery approach, positively 
valenced, refers to the focus of individuals on developing competence, expanding knowledge, task completion 
and understanding, learning, mastery, solving problems, and developing new skills. Mastery avoidance, 
negatively valenced, refers to the focus individuals have about striving to avoid misunderstanding or failing, 
avoid making mistakes or doing anything wrong or incorrectly from an intrapersonal perspective (Elliot & 
McGregor, 2001). Performance approach, positively valenced, is exemplified by learners’ concerns about how 
well they perform and how others perceive their behavior. It focuses on personal ability, a normative social 
comparison with others, and a desire for public recognition of performance. Performance avoidance, negatively 
valenced, focuses on avoiding normative competence, refers to low competence expectancies, fear of failure and 
avoidance of failure (Elliot & Church, 1997; Elliot & McGregor, 2001). 

In this study, it is expected that self-efficacy and the different subscales of goal orientation will influence tool 
use in relation to different tool characteristics. This expectation is based on empirical studies which have given 
some indication that different self-efficacy levels and goal orientation may affect tool use to a certain extent. 
However, the results of these studies are rather inconclusive and further research where tool characteristics 
interact with these motivational variables is has yet to be reported. For instance, it has been suggested that 
learners with high self-efficacy levels increase quantity of tool use; that means, they used tools more frequently 
(Wu, Lowyck, Sercu, & Elen, 2012). In contrast, Jiang and Elen (2010) indicated that high levels of self- 
efficacy affected the quantity of tool use negatively (i.e. the frequency of use) but the quality of tool use 
positively (e.g., the answers provided on adjunct questions). 

Regarding goal orientation, the effect has also not been consistent. It has been found a negative relationship 
between mastery approach goal orientation and quantity of tool use (time spent on tool) (Clarebout & Elen, 
2009) and (frequency) (Nesbit et al., 2006), also a positive relationship between performance goal orientation 
(approach and avoidance) on quantity of tool use (frequency) (Crippen, Biesinger, Muis, & Orgill, 2009). 
However, Huet, Escribe, Dupeyrat, & Sakdavong (2011) reported a significant negative correlation between 
performance goal orientation (approach and avoidance) and quantity of tool use (frequency of tool use). That 
means, learners with high performance approach and avoidance accessed the tools less frequently (Huet, et al., 
2011). No results from mastery avoidance goal orientation on tool use were retrieved in either study. 

TOOL CHARACTERISTICS 

Theoretically, the effectiveness of CBLE’s may be enhanced if appropriate cognitive tools are provided (Iiyoshi 
& Hannafin, 1998). Cognitive tools are support devices that scaffold the cognitive processes associated with 
learning or performing (Derry, Hmelo-Silver, Nagarajan, Chemobilsky, & Beitzel, 2006). By using cognitive 
tools, learners may be able to identify, locate and retrieve relevant information, to present relevant information 
they encounter, to structure and establish conceptual relationships, to manipulate information or to simplify their 
own learning, among others (Iiyoshi, Hannafin, & Wang, 2005). The distinctive role the learner may be able to 
execute is based on the type of cognitive tool. Adjunct questions, for example, are a cognitive tool that has the 
potential to guide students to organize, interpret (Elen & Louw, 2006) and influence what it is learned from the 
learning material (Hamaker, 1986). 
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Empirically , there is still lack of understanding in the influence of cognitive tool use in CBLE’s (Iiyoshi & 
Hannafin, 1998); more specifically the positive effects of adjunct questions on learning are not always 
consistent (e.g. Domisch & Sperling, 2006). Iiyoshi and colleagues (2005) point out that in order to attain 
positive effects of cognitive tools on learning, learners should receive more procedural assistance and sufficient 
guidance to use the tools. Therefore, aside from the type of tool, in this case cognitive tool, two additional tool 
characteristics come into sight. 

One of these tool characteristics is tool presentation (Schnotz & Heiss, 2009). Tool presentation refers to the 
level to which learners are induced to use the tools in the way intended by the designers/instructors. The 
presentation of a tool can be embedded or non-embedded (Clarebout & Elen, 2006). Tools are non-embedded 
when learners can decide whether to use the tools or not; on the other hand, when learners have no choice but to 
use the available tools then the tools are embedded (Clarebout & Elen, 2006). Having the tools embedded or 
non-embedded is in line with the learner control literature (Lawless & Brown, 1997) because non-embedded 
tools could benefit those learners that recognize the tool functionality and possess the motivational 
characteristics that would lead them to use the tool (Perkins' conditions: Perkins, 1985). However, embedded 
tools could benefit those learners unable to recognize the tool functionality and not motivated (Schnotz & Heiss, 
2009). 

Effects of the tool presentation, however, remain unclear. Greene and Land (2000) analyzed undergraduate 
learners’ use of cognitive tools: non-embedded (internet search engine machine) and embedded (guiding 
questions) while they worked on a project. The findings indicated that learners’ quality of tool use was low. 
First, they had difficulty using the searching tools properly. Their searches were too broad or irrelevant in many 
of the cases. Regarding the embedded questions, learners’ use of tools was suboptimal, that is, learners tended to 
omit questions and/or give superficial answers which did not allow them to benefit from the purpose of the tool. 
In more recent research (Clarebout, Horz, Schnotz, & Elen, 2010), 60 participants were divided into two 
conditions: with embedded tool sand with non-embedded tools (tools: graphs). The results indicated that the 
quantity of tool use in the embedded condition was significantly higher. This means that learners spent 
proportionally more time on graphs. However, the quality of tool use (interpretation of graphs) was significantly 
better in the non-embedded condition. 

Given that embedding or not embedding tools seems insufficient to assist the learner on the use of tools, 
instructional cues (Lee & Lehman, 1993) -another tool characteristic-, also addressed as advice (Clarebout & 
Elen, 2008) or pedagogical agents (Atkinson, 2002), among others have been implemented to increase tool use 
probabilities without ‘forcing’ the learner (Atkinson, 2002; Clarebout & Elen, 2008; Lee & Lehman, 1993). 

Instructional cues aim at making tool functionality more ‘discernible’ to the learner. This discemability may 
help the learner decide when and how to use or not to use tools (Lawless & Brown, 1997). These cues can either 
provide guidance to the learner during the task (Atkinson, 2002), highlight the use of tools that provide 
information that the learner has not accessed (Lee & Lehman, 1993), or provide information about the 
functionality of the tools, as well as, the learning benefits a tool can offer if used (Clarebout & Elen, 2008). 
Evidence has not only shown positive effects on quantity of tool use and performance (Atkinson, 2002; Lee & 
Lehman, 1993), but also mixed effects (Clarebout & Elen, 2008). These mixed effects showed that the advice 
was often ignored (Clarebout & Elen, 2008). Moreover, interactions between advice and learner characteristics 
have been found. 

The purpose of this study is to examine the role of self-efficacy and goal orientation in relation to tool 
presentation (non-/embeddedness of tool) and an instructional cue on the use of tools, namely adjunct questions. 
The instructional cues will be addressed, from now on, as explanation of tool functionality. The explanation of 
tool functionality followed the same focus as previous research (Clarebout & Elen, 2008) and will therefore 
provide information about the tools and their learning benefits. Tool use will be explored in a quantitative and a 
qualitative way. Lastly related to Perkins’ (1985) first condition of the ‘tool is there,’ this contribution addresses 
tool functionality by analyzing the effects of quantity and quality of tool use on performance. The addressed 
questions illustrated in figure 1 are: 

1. Does non-/embeddedness of tool influence quality and quantity of tool use? 

2. Does the (no) explanation of the tool functionality influence quality and quantity of tool use? 

3. Do self-efficacy and goal orientation influence tool use in relation to embeddedness and the 

explanation of the tool functionality? If so, how? 

4. Do quality and quantity of tool use influence performance? 
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Figure 1 . Schema of research questions. The schema is in abbreviated forms for simplicity and clarity. 


METHODOLOGY 
Participants and design 

Participants were one hundred and forty first-year university students of Educational Sciences which represented 
89.74 % of the whole population. Participation was a part of a course on Learning and Instruction. The average 
participants’ age was 18 (SD= 2.59), and the majority was female (94.3%). This was a quasi-experimental pre¬ 
post-test study with a 2 (non-/embedded tool) x 2 (with/without explanation of tool functionality) design. 
Participants in the study were randomly and equally assigned to one of four conditions (35 participants in each 
condition): a condition with embedded tools and the explanation of the tool functionality, a condition with non- 
embedded tools and the explanation of the tool functionality, a condition with non-embedded tools and no 
explanation and a condition with embedded tools and no explanation. 

Materials 

Learning Environment. The learning environment consisted of a hypertext which was developed using 
Macromedia Director. Two introductory pages were displayed before the hypertext. Participants filled in their 
personal data on the first page. In the second page a description was given of the structure of the text. 

The conditions with explained tool functionality had an extra paragraph on the second page with a detailed 
description of the functionality of the tool (adjunct questions) provided in the hypertext (figure 2). The 
explanation of the tool functionality read as follows: “Each question will explore a part of your knowledge. If 
you answer each of the questions, you will be able to find a clearer connection between the topic of the text and 
everyday life situations. By establishing this link, your knowledge will become more meaningful. If your 
knowledge is more meaningful, you will have more sources to answer the post-test in a more effective way.” 

After the introductory pages, the hypertext was presented. The hypertext was an environmental article titled 
Waarom water broodnodig is (Why water is essential)(Raes, Geerts, & Vanuytrecht, 2009). The article 
comprising 1,544 words was adapted in format for the task, and was therefore divided into five sections. Each 
section consisted of a page. After each section, an adjunct question (tool) was attached. In the non-embedded 
conditions, participants could access the tool through a button placed in the upper right of the page (figure 3). In 
the embedded conditions, the question was automatically displayed and had to be answered (figure 4). This text 
was not part of the curriculum and was chosen for this study as to raise environmental awareness. 

Tool: Adjunct question. The adjunct questions in this study were “inserted high-order post questions” (Hamaker, 
1986). This means that there was a question after each of the five sections of the hypertext, and the question 
elicited more than a simple collection of factual information as induced more complex cognitive processes in the 
learner (Hamaker, 1986). Ten adjunct questions were first designed by two researchers who agreed on selecting 
five of them. An example is shown in figure 4, and it is translated as follows: “Imagine that water was well- 
distributed on earth. Would food production still be a problem?” Example of another adjunct question (not 
shown in figure 4) is “According to the article, can the problem related to food production be solved by simple 
using less water?” 
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De volgende taak bestaat uit het lezen van een tekst. Bovenaan elke pagina kan je zien op welke pagina je je bevindt en 
hoeveel er nog overblijven. Na net lezen van elke sectie/passage van de tekst zal je een vraag krijgen 


Er zijn vijf secties in de tekst, en dus ook vijf vragen 



Elke vraag zal een deel van jouw kennis verkennen. Door ze allemaal te be 
een duidelijkere connectie te vinden tussen het onderwerp en het dagelijl 
teggen wordt jouw kennis meer betekenisvol. Doordat jouw kennis meer b 
middeterUiebben om de post-test op een meer effectieve manier te beanti 



Om de vragen te kunnen beantwoorden, is het zeer belangrijk dat je elke sectie van de tekst gedetailleerd en zorgvuldig 
leest, want het zal NIET toegelaten zijn notlties te nemen, Je mag wel zoveel keren teruggaan in de tekst als je wenst. 
Nadat je de taak op de computer hebt beeindigd, dien je een post-test op papier te beantwoorden, Deze post-test 
bestaat uit kennis en inzichtsvragen. 

Je mag teruggaan in de tekst indien je deze opnieuw wilt bekijken om de vragen te beantwoorden. 

Veel leesplezier! 


(Volgends pagina) 


Figure 2. Second introductory page with explanation of tool functionality. The explained functionality is 
illustrated inside the ellipse. Participants only saw the explanation boldfaced without ellipse. Non-explained 
functionality conditions did not have the boldfaced explanation. 

Self-efficacy. The questionnaire consisted of eight questions using elements from the Motivated Strategies for 
Learning Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1991) and the Self- and Task- 
Perception Questionnaire (STPQ) (Lodewyk & Winne, 2005). The questions were adapted to the context of tool 
use and translated to Dutch using the translation/back translation method in order to avoid semantic problems. 
This questionnaire has previously been used and shown high reliabilities (Jiang & Elen, 2010). Examples of the 
items are “I believe I will receive an excellent grade in the test over the text” or “I’m certain I will master the 
skills necessary for learning this text.” A six-point Likert scale was employed where one indicated total 
disagreement and six indicated total agreement. The reliability obtained was a=.84. 

Goal orientation. Goal orientation was measured by merging two questionnaires of Elliot et al. (Elliot & Church, 
1997; Elliot & McGregor, 2001). The initial questionnaire of Elliot et al.(Elliot & Church, 1997) only measured 
three dimensions of goal orientation (mastery approach, performance avoidance and performance approach). 
The revised questionnaire (Elliot & McGregor, 2001) incorporated mastery avoidance and mastery approach 
making a 2 x 2 framework of goal orientation (performance avoidance, performance approach, and mastery 
avoidance and mastery approach). Participants also responded on a six-point Likert scale. One indicated totally 
disagree and six totally agree. The reliabilities were above an a value of .70: Performance approach (a=.93) and 
avoidance (a=.83), mastery approach (a=.84) and avoidance (a=.71). 

Tool use: Quantity- Time spent on tool and frequency of tool access -. The time spent on the tool was 
considered as a measure for quantity of tool use in all four conditions. Log files were kept in a Microsoft Access 
database which contained the learner’s identity and the time spent on the tools in seconds. Another indicator of 
quantity of tool use was considered but only in the non-embedded conditions. This was the clicks made to 
access the tool, namely, frequency of tool use. This data was also obtained through the log file which recorded 
the amount the times the learners click on the button to access the tool, namely adjunct question. 
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Vraag 


Pagina 5/15 


( Vraag 1 ) 


( Vorige pagina ) 


fvolgende pagina ] 


Figure 3. Screenshot of non-embedded conditions. After learners read a section they had a blank page with a 
button that read ‘vraag’ (question). Learners had the option to access the tool or not by clicking on that button. 

Tool use: Quality. Answers on the adjunct questions were individually logged as text files. The correctness of 
the answers was used as the quality indicator of tool use. Participants could obtain from zero to three points per 
answer on each adjunct question. Zero indicated no answer, an incorrect answer or an incorrect answer without 
any rationale. One point was given to a correct answer without rationale and/or explanation. Two points were 
given to a correct answer but with little or no rationale. Three points reflected correct answer with a high 
rationale and well thoroughly provided answer. Participants could obtain in total 15 points. The answers were 
scored by three different researchers. The inter-rater reliability using Intraclass correlation (ICC) was 
considered to be appropriate if the value was above .80 (Shrout & Fleiss, 1979). The ICC for grading the adjunct 
questions showed outstanding agreement among the three raters (ICC= .99, p <.001). Two out of the three raters 
were social researchers who were not involved in the present study. 

Learning outcomes. Prior knowledge was measured by a pre-test (ni tems =9) in order to assess possible differences 
among conditions. Performance was assessed using a post-test (ni tem s = 13). Both tests were designed by three 
researchers, authors of this paper, two of which had experienced in the design of these types of materials. 
Consequently, the design of the pre- and post-test followed similar criteria used in previous studies (Clarebout, 
et al., 2010). The nine pre-test questions were multiple choice questions which explored learners’ factual 
knowledge related to the topic of the hypertext. Each question had four different choices. Two examples of such 
questions are as follows: “How much water in average does a Belgian use per year?” and “What does water 
footprint refer to?” Each correct question was worth a point. Therefore, learners could obtain a maximum nine 
points. In the post-test, learners could obtain up to 16 points. It included seven multiple -choice factual 
knowledge questions (e.g., what does FAO stand for?) and six insight questions among which three were open 
questions (e.g., what did the author imply with “more crop per drop”?) and three were flll-in-the-blank questions 

with optional words (“In the next 50 years the demand of food will_”). Every correct answer was 

given one point except for the open questions which could be graded from zero to two. Learners obtained two 
points for correct answer and a deep rationale answer, one point for a correct answer weak rationale answer and 
zero points for no rationale, no answer or incorrect answer. The inter-rater reliability for the scoring procedure 
of the open questions among the three raters was found to be outstanding ( ICC= .97, p <.001). As in the pre-test 
two out of the three raters were researchers from other study domains. 


Copyright © The Turkish Online Journal of Educational Technology 


335 










TOJET: The Turkish Online Journal of Educational Technology - April 2013, volume 12 Issue 2 


Vraag 


Pagina 5/15 


1. Beeld je in dat het water goed verdeeld is over de aarde. Zou voedselproductie nog steeds een probleem zijn? 


□ 


( Vorige pagina ) 


(Volgende pagina) 


Figure 4. Screenshot of embedded conditions. Embedded conditions accessed the tool after clicking the button 
to go to the next page. The darkened space was provided to write the answer. 


Procedure 

The study was carried out in two steps. First, the self-efficacy and goal orientation questionnaires were 
administered to all participants during their Learning and Instruction class. Answering the questionnaires was a 
pre-requisite to attend the second part of the experiment; also, participants had to register themselves in a 
learning platform where they selected a session time in which they could attend. The second part was spread 
over different sessions in a computer laboratory with a maximum of 20 participants assigned to different 
conditions per session (maximum 40 minutes duration). 

In the second part of the experiment, first, participants answered the pre-test on paper. Afterwards, they were 
introduced to the CBLE and started reading the hypertext. When the participants finished, they raised their hand 
and waited in silence until the researcher approached them. Once the researcher, who was conducting the 
research session, approached them, she checked the log files and made sure the learners’ answers to the adjunct 
questions, time and clicks were recorded. If everything was appropriately saved, the researcher closed the 
program and handed the post-test on paper to be answered. Once participants completed the post-test, they were 
given proofs of the attendance and left the computer lab. 

Data analyses 

First, a MANOVA was used to check whether the conditions differed with respect to goal orientation, self- 
efficacy the pre-test (prior knowledge) and the post-test (performance). Condition was the independent variable 
and the learner characteristics, pre- and post-tests were the dependent variables. If groups showed a difference 
among conditions, then any of these variables were to be considered covariates in the further analyses. 

Research questions 1& 2: Does non-/embeddedness of tool influence quality and quantity of tool use? 
Does the (no) explanation of the tool functionality influence quality and quantity of tool use? And 
Research question 3: Do self-efficacy and goal orientation influence tool use in relation to embeddedness 
and the explanation of the tool functionality? If so, how? 

For questions 1 and 2, descriptive statistics were run between conditions and quantity and quality of tool use to 
see the effect of condition, more specifically, non-/embeddedness and (no) explained functionality on tool use. 
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Next, to answers research questions 1,2 and 3, a MANOVA was conducted with all the conditions; self- 
efficacy, performance approach, performance avoidance, mastery approach and mastery avoidance respectively 
as independent variables to see their effect on quality and quantity (time spent on tool) of tool use (dependent 
variables). For the effect of frequency of tool use (quantity), only the non-embedded conditions were 
considered. It is important to mention that all participants were considered given that all of them accessed the 
tools. Flerein, a factorial ANOVA was conducted in which non-embedded conditions; self-efficacy, performance 
approach, performance avoidance, mastery approach and mastery avoidance were independent variables and 
quantity of too use (frequency) the dependent variable. 

Research question 4: Do quality and quantity of tool use influence performance? 

To answer the last research question, first, correlation analyses were conducted in order to see any possible 
correlations among tool use measurements. Second, regression analyses were conducted. The first regression 
with performance as dependent variable and tool use, namely quality of tool use and time spent on tool as 
independent variables. The second regression was done with frequency of tool use (quantity) as independent 
variable and only in the non-embedded conditions. 

RESULTS 

The MANOVA indicated that the mean of each condition regarding self-efficacy, the scales of goal orientation, 
the pre-test (prior knowledge) and the post-test (performance) did not significantly differ between conditions, 
Wilks’ lambda = .84; F (21,373)= 1. 12, p=. 32 rj 2 partia i =.06. Therefore in the further analyses, none of these 
variables were considered as covariates. 

Research questions 1& 2: Does non-/embeddedness of tool influence quality and quantity of tool use? 
Does the (no) explanation of the tool functionality influence quality and quantity of tool use? 

The descriptive statistics in Table 1 show that both embedded conditions and non-embedded conditions used all 
tools. The MANOVA indicated a significant effect of condition on quality and quantity of tool use (time spent 
on tool) Wilks ’lambda=0. 632, F (6, 230)=9.89, p<.001. rj 2 'partial =-20. The separate ANOVA’s on the outcome 
variables confirmed this significance: quality of tool use F (3, 116) = 4.22, p<.01, ij 2 partia i =.10 and quantity F (3, 
116) =17.04, jcK.001, r\ 2 pania i =.31. Post hoc analyses using the Tukey post hoc criterion for significance 
indicated that the time spent on tools was significantly higher in the embedded condition with explained 
functionality (p <.005); and without explained functionality (p<. 05) in comparison with both non-embedded 
conditions. However, the embedded condition with explanation was significantly higher than the embedded 
condition without explanation (p<.001). 


Table 1. Descriptive statistics of quantity (time spent in seconds) and quality (answers on adjunct questions) of 
tool use across conditions. Non-embedded conditions show also tool use frequency, that is, clicks learners made 

to access tool. 


Condition 

Tool use 

N 

Mean 

SD 

Embedded, explained functionality 

Quality (max. 16 points) 

34* 

9.12 

2.17 


Quantity: Time spent (seconds) 

35 

793.03 

370.85 


Performance (max. 13 points) 

35 

11.26 

1.67 

Embedded, non-explained functionality 

Quality (max. 16 points) 

35 

8.43 

1.82 


Quantity: Time spent (seconds) 

35 

583.31 

235.23 


Performance (max. 13 points) 

35 

11.26 

2.10 

Non-embedded, explained functionality 

Quality (max. 16 points) 

35 

9.77 

2.30 


Quantity: Time spent (seconds) 

35 

402.17 

143.04 


Quantity: Frequency (clicks) 

35 

16.77 

9.38 


Performance (max. 13 points) 

35 

11.71 

2.08 

Non-embedded, non-explained functionality 

Quality (max. 16 points) 

35 

9.94 

1.51 


Quantity: Time spent (seconds) 

35 

398.11 

175.95 


Quantity: Frequency (clicks) 

35 

18.09 

10.77 


Performance (max. 13 points) 

35 

11.06 

2.01 


* Data from one participant was lost 
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The descriptive statistics also revealed that non-embedded conditions tool usage quality was higher than the 
embedded conditions. The Tukey post hoc narrows down this finding and points out that non-embedded 
condition with explained functionality (p<.05) and without explained functionality (p<.005) were significantly 
different only with the embedded without explained functionality. However, significant differences between the 
embedded condition with explanation and without explanation on quality of tool use were not retrieved in the 
post hoc test {p=. 55). 

Although the descriptive statistics in the non-embedded conditions suggested that participants accessed tools 
relatively more frequently in the condition without the explanation of tool functionality (M = 18.09) than in the 
condition with explained functionality (M = 16.77), the ANOVA indicated that such difference was not 
significant F (1, 58) =.19,/?=.67. r] 2 par tiai =.00. 


Research question 3: Do self-efficacy and goal orientation influence tool use in relation to embeddedness 
and the explanation of the tool functionality? If so, how? 

Performance approach had a significant direct effect on tool use, Wilks lambda= 0.95; F (2,115) =3.09 /K.05. 
r] 2 partiai = -05- Univariate ANOVA’s revealed that performance approach did influence quality of tool use F (2, 
95) = 3.88, p=. 05, tj 2 ^,,1 =.03 but did not affect quantity of tool use (time). The particular effect of performance 
approach on quality of tool use was mainly positive meaning that learners with high levels of performance 
approach used tools more qualitatively in comparison with learners with low levels of perfonnance approach 
(Figure 5). 



8 . 0 - 


- 1 - 1 - 

Low High 

Perfomance Approach 

Figure 5. Effects of performance approach on quality of tool use. 

Another variable that showed an effect on tool use was master avoidance, but this in interaction with condition. 
This effect could only be observed using Roy’s statistics. Roy’s largest root =0. 88; F (3,116) =3.40 p=. 02. 
rj 2 P ai-tiai = -Q%- The further univariate ANOVA’s confirmed that master avoidance in interaction with condition did 
influence quality of tool use F (3,116) = 3.19, /?<.05, rj 2 parlia i =.08 but did not affect quantity of tool use (time). 
Both low and high levels of master avoidance used tools more qualitatively in both non-embedded conditions 
(with explained and non-explained functionality) than in both embedded conditions. The interaction effects can 
be observed in more detailed in table 2 and figure 6 which show the estimated marginal means given that means 
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of unequal sizes were compared. The MANOVA revealed no further significant (interaction) effects of 
performance avoidance, master approach and self-efficacy on tool use. 

Table 2. Estimated marginal means, standard errors for quality of tool use of the different conditions, and the 

low and high levels of master avoidance. 


Condition 

Mastery 

Avoidance 

levels 

N 

Estimated 

Marginal 

Mean 

SE 

Embedded, explained functionality 

Low 

18 

8.94 

0.57 


High 

17 

8.46 

0.61 

Embedded, non-explained functionality 

Low 

25 

8.09 

0.45 


High 

10 

9.14 

0.71 

Non-embedded, explained functionality 

Low 

25 

9.11 

0.45 


High 

10 

11.18 

0.69 

Non-embedded, non-explained functionality 

Low 

22 

10.55 

0.47 


High 

13 

9.02 

0.67 


12 - 


a> 

in 

3 



- 1 - 1 - 1 - 1 - 

Embedded Embedded Non-embedded Non-embedded 

Tool/Explained TooliNon-explained Tool/Explained Tool/Non-explained 
Functionality Functionality Functionality Functionality 


Condition 


Mastery 

Avoidance 

- - Low 
High 


Figure 6. Interaction effects of mastery avoidance and condition on quality of tool use. 

The ANOVA analysis in the non-embedded conditions (jV= 70 each) showed that no variable in interaction with 
condition had a significant impact on frequency of tool use. Only after it was controlled for condition, self- 
efficacy had a significant effect on quantity of tool use (frequency) F (1, 58) = 8.45, p<. 01, ij 2 parlia t =.13. This 
effect was negative, meaning that the lower the self-efficacy in learners the more frequent they accessed the 
tools (Figure 7). 
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Figure 7. Self-efficacy on frequency of tool use. 


Research question 4: Do quality and quantity of tool use influence performance? 

A significant negative correlation between frequency of tool use and quality of tool use was observed (r= -.26, 
/?<.05) but none of these tool use measurement affected performance significantly. That means that, as 
illustrated in Table 3, only quantity of tool use, namely time spent on the tool, seemed to be the only aspect of 
tool use that affected perfonnance. Although all learners in all conditions used the tools, it seemed that only 
time spent on tool had a significant impact. 

Table 3. Regression analyses on the effects of tool use on performance. 



Performance 

all conditions* 



Perfonnance 

non-embedded 

conditions** 


B 

SEB 

P 


B 

SEB 

P 

Step 1 

(Constant) 

10.57 

0.37 


(Constant) 

11.67 

0.50 


Quantity (time on tools) 

0.001 

0.001 

0.19*** 

Quantity (frequency) 

-0.02 

0.03 

-0.08 


Step 2 

(Constant) 

9.23 

0.85 

Quantity (time on tools) 

0.001 

0.001 0.20*** 

Quality tool use 

0.14 

0.08 0.15 


*R 2 = .19 for Step 1, p<.05. AR 2 = ,24 for Step 2,p=08. 
**R 2 = .01 for Step 1, p.52 (one step only) 

***p <,05 
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DISCUSSION & CONCLUSION 

This contribution was aimed at gaining more insight into learners’ quantitative and qualitative use of tools by 
exploring the impact of tool characteristics (non-/embeddedness, with/without the explanation of tool 
functionality) and motivational characteristics (self-efficacy and goal orientation) on performance in a CBLE. 
Some learners had to answer the questions (embedded conditions) whereas others could choose whether to use 
the questions or not (non-embedded conditions). Within these conditions the explanation of the tool 
functionality was added in two of the four conditions (one embedded and one non-embedded condition). 



Figure 8. Summary of significant results from present study. 


One of the results illustrated in figure 8 is that learners in both embedded conditions spent proportionally more 
time on the tools than both of the non-embedded conditions. Moreover, the results from the Tukey post hoc test 
also suggested that the explanation of tool functionality affected the time spent on the tool. This is because the 
embedded condition with explained functionality was significantly higher (in time) than the embedded condition 
without explanation. Additionally, learners in the non-embedded conditions with and without explained 
functionality had better results in the quality of tool use which means they answered the adjunct questions more 
thoroughly than the embedded condition without explained functionality, but not more than the one with 
explained functionality. 

The findings on the relationship between tool presentation (non-/embedded tool) and quantity (time) and quality 
of tool use are similar to previous research (Clarebout, et al., 2010) where participants in the embedded 
condition used tools more but less qualitatively than participants in the non-embedded condition. Therefore 
embedded tools may be in the end a straightforward solution to avoid non-use of tools, but non-embedded tools 
may ensure the quality of tool use. This discrepancy brings to light the two standpoints of the learner control 
debate (see: Lawless & Brown, 1997). One point supports the idea of giving the learners control over their 
learning, as for example, in the component display theory (Merrill, 1983) whose one of the main principles is 
that learners should be given control over the number of instances or practice items they receive. The second 
point claims that learning is not effective in environments with little control over learning. However given that 
in the present study quality of tool use did not impact performance, possibly in the present experimental setting 
embedding tools may be more optimal. 

The finding on the explanation of the tool functionality, on the other hand, seems inconclusive. While it 
provides evidence that (a) explained tool functionality may augment time spent on the tool when tools are 
embedded, and that (b) non-embedded tools (with and without explanation) and embedded tool with explained 
functionality may increase quality of tool use, it may also suggest that the non-/embeddedness has a stronger 
effect on tool use than the explanation itself. One possible reason why the explanation of the tool functionality 
was not significant could be attributed to the nature of the explanation itself. The explanation of the tool 
functionality was presented only once, namely at the beginning of the text. Having the explained tool 
functionality at one moment was probably too ‘minimal’; moreover, the explanation to use tools was perhaps 
not formulated in an encouraging way for these types of learners. These two reasons may have lead to an 
ineffectiveness on the explanation of the tool functionality (Kirschner, Sweller, & Clark, 2006). This finding 
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may also explain why quality of tool use did not affect performance (further discussion below). Hence to 
increase the possibility of having a positive effect of explained functionality on the use of tools, and 
consequently on performance, a specific more guided instruction on the use of tools should be provided 
(Kirschner, et al., 2006). It is proposed that this could be done by increasing the amount of times the explanation 
of the tool functionality is present, for example, before accessing the tool and probably also by adapting the 
explanation in a way that encourage learners towards a more optimal tool use. 

While a direct effect of the explanation of the tool functionality on tool use could not be confirmed, it could also 
be argued that the partial effect found through the post hoc tests could be mediated by affective characteristics. 
In broad terms, research has suggested that affective characteristics may impact use and effects of technology 
(Proost, Elen, & Lowyck, 1997). Moreover, it has been indicated that how learners perceive the explanation of 
the tool functionality may impact on the effect of the explanation (Clarebout & Elen, 2008). This means that the 
emotional reactions towards the tool were probably different among conditions (with/without explanation), 
making the emotional reactions towards the tool more positive in the conditions with the explained 
functionality. Further study research on tool use introducing qualitative approaches could possibly widen the 
perspectives on tool use, for example, through observations. 

This contribution also revealed that two aspects of goal orientation (figure 8) influenced tool use. Specifically, 
high levels of performance approach positively influenced quality of too use. This means that performance 
approach learners, namely learners with high concerns about how well they perform and how others perceive 
them, answered the adjunct questions more thoroughly. Crippen and colleagues (2009) also found that 
performance approach was positively related to tool use , but Huet, et al. (2011) revealed opposite results. In 
both studies (Crippen, et al., 2009; Huet, et al., 2011) the measures of tool use was based on the times learners 
accesses the tools (frequency). Our result is more in line with the results obtained by Crippen and colleagues 
(2009) because of the positive relationship found, and also add to the literature by suggesting that performance 
approach is also positively related of the quality of tool use. This result may also suggest that that the quality of 
tool use was better in learners with high performance approach levels because quality of tool use was viewed as 
a simpler task to demonstrate success than spending more time on the tool or accessing it more (Crippen, et al., 
2009). 

Considering the definition of mastery avoidance, learners that strived at avoiding misunderstanding, failing or 
making mistakes from an intrapersonal perspective showed no direct effect on tool use. Only when interacting 
with condition an effect of master avoidance was found to influence the quality of tool use (students’ answers 
given to the adjunct questions). Learners with high and low mastery avoidance levels in non-embedded 
conditions showed more quality of tool use. These findings are related to the results obtained from tool 
presentation and explanation of tool functionality on tool use which means they could confirm the fact that both 
non-embedded conditions use tools more qualitatively. Moreover, an indirect influence of the interaction 
between mastery avoidance and condition on frequency of tool use could also be suggested given that frequency 
of tool use and quality of tool use were negatively correlated. This influence would be negative, which implies 
that learners that avoided misunderstanding or making mistakes accessed tools less frequently in the non- 
embedded conditions. However no further implications could be made given that neither frequency of tool use 
nor quality of tool use influenced performance. In general, the finding showing the interaction effects of 
mastery avoidance could sustain the theory of aptitude-treatment interaction (Cronbach & Snow, 1977) and 
bring to light the role of learners characteristics in relation to tool characteristics, in this case the presentation of 
tools in CBLE’s. 

For the other aspect of quantity of tool use-frequency-, no interaction effects between learner characteristics and 
conditions were found. However, significant direct effect of self-efficacy, on frequency of tool use was found 
(figure 8). The further analysis suggested that high self-efficacy levels influenced negatively the frequency of 
tool use and hence -due to the found correlation- indirectly had a positive influence on the quality of tool use. 
Considering that self-efficacy is defined as “the beliefs in one’s ability to plan/execute a behavior” (Bandura, 
1997), if people, in this case learners, believe they have a great ability to organize and execute the course of 
actions required to succeed and reach a certain goal, then they will be capable of adapting their use of tools. 
Moreover, this finding might also be related to the optimization stage for tool use (Iiyoshi & Hannafin, 1998), as 
to the third condition of Perkins (1985): Learners, with a high level of self-efficacy -a motivational 
characteristic-use the tools, attempted to optimize the tool use by decreasing the frequency of tool use. 

In this study, it was assumed that quantity and quality of tool use would influence performance. Confirming this 
would fulfill Perkins’s first condition (1985) of the tool is there, more specifically, the tool is functional for the 
learner (Clarebout, et al., 2010). The findings may raise the question on whether the present tool was fully 
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functional as they revealed that only the time spent on the tool (quantity of tool use) influenced performance and 
neither frequency of tool use nor quality of tool use revealed a significant impact. 

Additionally, the fact that only time spent on tool had a positive effect on performance also question whether it 
is appropriate to explore tool use from a quantitative (time and frequency) and qualitative perspective in a single 
study. Literature on tool use has explored one of the three (frequency:Crippen, et al., 2009) obtaining positive 
results on performance. Other studies have focused on two of them (time & frequency: Elen & Louw, 2006; 
Viau & Larivee, 1993) . Elen & Louw (2006), for instance, saw only partial effect of frequency of tool use on 
performance, while Viau & Larivee (1993) showed effects of both time and frequency on performance but only 
in one of the two tools. Few studies have explored the three of them (Clarebout, et al., 2010; Jiang & Elen, 
2011). Clarebout and colleagues (2010) only found significant positive effects of time and quality on tool use on 
performance; Jiang and Elen (2011) had positive effects of quality of tool use on performance but negative 
effects of frequency of tool use on performance, partial effects of time on tool on performance were revealed. 
Other studies have not analyzed the effects of tool use on performance (quality: Greene & Land, 2000; 
frequency: Huet, et al., 2011). 

The results in the present investigation could also question whether quantity and quality of tool use are separate 
measurements for tool use, given that a correlation between frequency of tool use and quality of tool use was 
found. It is also questioned whether the time spent on the tool could be indirectly related to what is conceived as 
the quality of tool use. That is to say, the time spent on the tool -adjunct question- could also be an indicator of 
quality of tool use because learners possibly took more time thinking on the question than actually writing down 
a thorough answer. Learners may see the value of thinking about the questions but not the value of writing all 
their thoughts down and making the answers look ‘perfect.’ 

Overall, the results reveal the complexity involved when using tools in CBLE’s. They also add to our 
understanding of the relationships between learner and tool characteristics on tool use in university students. In 
general, the findings suggest that goal orientation, self-efficacy and tool characteristics interact and affect the 
quantity and quality of tool use; and quantity of tool use affects performance. 
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